Talking to Robots

A team of  Idaho State University students and faculty has one year to create a system that will enable programmers wearing augmented reality glasses to communicate directly with robots.

There are seven ISU undergraduate students, three graduate students and two mechanical engineering faculty who are working with the robotic systems integration company House of Design in Nampa to design and develop an augmented reality platform for robotic systems design and interaction.

The project started in January, funded by a $162,606 grant from the Idaho Global Entrepreneurial Mission (IGEM) Council, which is administered by the Idaho Department of Commerce. They have until the end of 2019 to get the project done.

“The project has to be finished and we have to deliver to House of Design by December,” said Alba Perez, co-principal investigator, ISU associate professor and chair of mechanical engineering. “The system we’re creating will have to integrate with the HoloLens (augmented reality glasses created by Microsoft) so users will be able to communicate with the robots to send and receive information. We have to make the programming and the setup very natural and easy.”

Omid Heidari, a Ph.D. student from Iran who is working on the project, said 12 people involved with the study have been divided in four groups to focus on different aspects of the project. There are two major challenges, managing people and creating new knowledge.

“We need  every group to do their task on time and hit deadlines. That’s one part of it, ”Heidari said. “The other part is the technology is new and we need to do a lot of research. Everyone has to do a lot of research. Then we decide for each step of the project what we need to do and what should be done.”

The grant’s other principal investigator is Mechanical Engineering Professor Marco Schoen in the College of Science and Engineering. ISU will work with House of Design CEO Shane Dittrich. They’ll be completing the work at ISU’s Measurement and Control Engineering Research Center. An industrial robot worth about $35,000 is being provided to ISU for the project by House of Design. House of Design does custom automation of robots that are used in industrial settings.

Students working on the project expressed their enthusiasm.

“I think it is cool,” said Michael Kelly, a senior mechanical engineering student from Cape Cod, Massachusetts. “I think this is something that people haven’t even done professionally, so to be able to get to do it on the educational side of things is even cooler. They don’t even have the virtual reality interaction with the robots, yet.”

Shovan Chowdhury, a master’s mechanical engineering student from Bangladesh, came to ISU just to work on this project.

 “I am really interested in this project because this augmented reality is a new topic that no one else is doing big research on,” Chowdhury said. “We can start a big thing and it will be great and there may be many more opportunities that come in the future because of this project.”

Perez said she envisions a programmer wearing Microsoft Hololens augmented reality glasses will be able to see the real world, but then when they’re within a certain distance they can communicate wirelessly with the robot. The augmented reality glasses may feature menus off to the side that programmers can see and then select options. It is possible the augmented reality system could use voice commands for communications between the programmers and the robots.

Robotic systems are being deployed more and more in many industrial processes, and House of Design is a leader company in this field, which was named 2017 Innovative Company of the Year in Idaho, Perez said. This project has grown out of ISU’s expertise in developing a virtual reality system to assist with arm rehabilitation and Heidari’s internship with House of Design.

“We expect House of Design to benefit from having a very natural application to interact with the robots,” Heidari said. “Looking at the future, applications in augmented reality may expand to many activities in industry, health care and the environment in  which robotics may also be involved”